System and method for displaying an image indicating a positional relation between partially overlapping images

ABSTRACT

A camera system includes a display monitor which displays an image of an object, taken by an optical unit, on a screen of the monitor. A reading unit reads a preceding image and a current image among a plurality of partially overlapping images, from a memory device, the preceding image and the current image containing a common element. A determining unit determines a positional relation between the preceding image and the current image based on a common pattern derived from the common element in the two adjacent images read by the reading unit. A displaying unit displays an image indicating a boundary of the preceding image on the screen of the monitor at a shifted position according to the positional relation determined by the determining unit, with the current image concurrently displayed on the screen of the monitor.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention relates to a camera system which electronicallystores an image of an object and displays the image on a displaymonitor.

(2) Description of the Related Art

Generally, to achieve an adequately high level of resolution of an imagecaptured by using a digital camera or a video camera, it is necessary touse a zoom-up function of the camera or move the camera close to anobject to be imaged. This makes it difficult to obtain an image coveringa wide angle related to the object. To capture an image covering a wideangle related to the object, it is necessary to use a zoom-down functionof the camera or move the camera away from the object. However, thismakes it difficult to obtain an image with a high level of resolution.

In order to obtain a wide-angle image with a high resolution from anobject, a divisional shooting method has been proposed. In thedivisional shooting method, a plurality of partially overlapping imagesare successively shot so as to cover a wide angle related to the object,and they are synthesized to create a composite image with an adequatelevel of resolution.

As disclosed in Japanese Published Utility Model Application No. 8-4783,an image processing device which is capable of combining a plurality ofpartially overlapping images together to create a composite image isknown.

To effectively carry out the divisional shooting method, it is necessarythat, after a preceding image is taken and before a current image istaken, the user stop movement of an optical axis of the camera at anappropriate position where an overlapping portion of the two adjacentimages is appropriate for subsequently producing a composite image fromthe images. However, in order to meet this requirement, a conventionaldigital camera requires a special adapter. If such an adapter is notused, it is difficult for the conventional digital camera to effectivelycarry out the divisional shooting method. In a case of the conventionaldigital camera with no special adapter, there is a possibility that nooverlapping portion exists between the two adjacent images or a toolarge overlapping portion be produced between the two adjacent images.If the overlapping images with undesired overlapping portions areobtained through the divisional shooting method, it is difficult toeffectively combine or synthesize the images together to create acomposite image.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a camera system whichdisplays an image indicating a positional relation among partiallyoverlapping images, and enables an operator to easily and effectivelycarry out a divisional shooting process.

Another object of the present invention is to provide a divisionalshooting method which displays an image indicating a positional relationamong partially overlapping images on a screen of a monitor during adivisional shooting mode of a camera system.

The above-mentioned objects of the present invention are achieved by acamera system which comprises: a display monitor which displays an imageof an object, taken by an optical unit, on a screen of the monitor; areading unit which reads a preceding image and a current image among aplurality of partially overlapping images, from a memory device, thepreceding image and the current image containing a common element; adetermining unit which determines a positional relation between thepreceding image and the current image based on a common pattern derivedfrom the common element in the two adjacent images read by the readingunit; and a displaying unit which displays an image indicating aboundary of the preceding image on the screen of the monitor at ashifted position according to the positional relation determined by thedetermining unit, with the current image concurrently displayed on thescreen of the monitor.

The above-mentioned objects of the present invention are achieved by adivisional shooting method for a camera system in which at least two ofpartially overlapping images of an object, taken by an optical unit, aredisplayed, comprising the steps of: reading a preceding image and acurrent image among the partially overlapping images, from a memorydevice, the preceding image and the current image containing a commonelement; determining a positional relation between the preceding imageand the current image based on a common pattern derived from the commonelement in the two adjacent images; and displaying an image, indicatinga boundary of the preceding image, on a screen of a display monitor at ashifted position according to the positional relation determined by thedetermining step, with the current image concurrently displayed on thescreen of the monitor.

In the camera system of the present invention, a positional relationbetween the preceding image and the current image is determined based ona common pattern derived from the common element in the two adjacentimages. The operator can easily carry out a divisional shooting mode ofthe camera system by viewing both the current image and the imageindicating the positional relation between the partially overlappingimages on the screen of the monitor. The positional relation between thepreceding image and the current image is clearly noticeable to theoperator by viewing the positional relation image on the screen of themonitor together with the current image while the camera is panned in adesired direction. Therefore, the operator easily stops the movement ofthe optical axis of the camera at an appropriate position by viewing thepositional relation image on the screen of the monitor, and turns ON ashutter switch to store the current image.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects, features and advantages of the present invention willbecome more apparent from the following detailed description when readin conjunction with the accompanying drawings in which:

FIG. 1 is a block diagram of a preferred embodiment of a camera systemof the present invention;

FIG. 2 is a flowchart for explaining a first example of a divisionalshooting process performed by a processor of the camera system;

FIG. 3A and FIG. 3B are diagrams showing an image which is displayed ona screen of a display monitor when the camera is moved in a givendirection;

FIG. 4 is a flowchart for explaining a second example of the divisionalshooting process performed by the processor of the camera system;

FIG. 5 is a flowchart for explaining a third example of the divisionalshooting process performed by the processor of the camera system;

FIG. 6 is a flowchart for explaining a fourth example of the divisionalshooting process performed by the processor of the camera system;

FIG. 7 is a flowchart for explaining an image storage process performedby the processor of the camera system when a shutter switch is turnedON; and

FIG. 8A and FIG. 8B are diagrams for explaining a determination of apositional relation between partially overlapping images in thedivisional shooting process according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A description will now be given of the preferred embodiments of thepresent invention with reference to the accompanying drawings.

In order to carry out a divisional shooting process, the presentinvention utilizes a method and a system for determining a positionalrelation between partially overlapping images based upon a commonpattern in an overlapping portion of the images. The method and thesystem are disclosed, for example, in U. S. patent application Ser. No.08/807,571 filed on Feb. 27, 1997 and U. S. patent application Ser. No.08/966,889 filed on Nov. 10, 1997, both assigned to the applicant of thepresent application. The contents of these co-pending applications arehereby incorporated by reference.

FIG. 1 shows a preferred embodiment of a camera system of the presentinvention. One example of the camera system of the present invention isa digital camera.

As shown in FIG. 1, the camera system of the present embodiment includesan optical unit 10. The optical unit 10 has an image pickup device 12, alens (not shown), and a lens positioner (not shown). The image pickupdevice 12 is comprised of a charge-coupled device (CCD). The imagepickup device 12 converts light incident from an object into anelectrical signal, or an image signal indicative of an input image ofthe object or the scene. The lens positioner mechanically positions thelens of the optical unit 10 at a desired distance from the object alongan optical axis of the lens. Hereinafter, the lens of the optical unit10 will be referred to as the camera.

In the camera system of the present embodiment, a lens positioneractuator 14 actuates the lens positioner of the optical unit 10 so thatthe lens is positioned at a desired distance from the object along theoptical axis of the lens. An operation part 16 is an operation part ofthe camera system of FIG. 1, which includes a mode selection switch 18,a shutter switch 20, and other control switches (not shown). An operatorcan manipulate one of such switches of the operation part 16 so as toselect one of operational modes of the camera system or to release theshutter of the camera system.

In the camera system of the present embodiment, a video control unit 24converts the signal from the image pickup device 12 into a digitalsignal, processes the digital signal to produce a frame of the inputimage, and stores the frame in a frame buffer 25. The frame or imagedefined in the frame buffer 25 is a pixel map that has an array of pixeldata, each indicating an intensity (and/or a color value) for a positionof a corresponding one of the picture elements, or pixels, in the image.The video control unit 24 displays the image defined in the frame buffer25 on a liquid-crystal display (LCD) monitor 27, accessing the framebuffer 25 as frequently as a scan rate of the monitor 27. The monitor 27has a display screen 27A, and the image defined in the frame buffer 25is displayed on the screen 27A of the monitor 27 by the video controlunit 24.

The video control unit 24 further includes a frame buffer 26 in additionto the frame buffer 25. The frame buffer 26 stores auxiliary dataindicative of a peripheral boundary 27B (which will be described later)corresponding to the image defined in the frame buffer 25. The videocontrol unit 24 displays the peripheral boundary 27B, indicated by theauxiliary data defined in the frame buffer 26, on the screen 27A of themonitor 27, accessing the frame buffer 26 at the same time as the framebuffer 25. Hence, the image defined in the frame buffer 25 and theauxiliary data defined in the frame buffer 26 are synthesized so thatthe image with the peripheral boundary 27B is displayed on the screen27A of the monitor 27 in an overlaid manner. The auxiliary data definedin the frame buffer 26 includes a frame number to identify a capturedimage among a plurality of partially overlapping images, which will bedescribed later. Further, the auxiliary data may further include imagedata of a displacement vector or a direction of the optical axis of thecamera, which will be described later.

In the camera system of the present embodiment, an image memory 28 is astorage device which stores an image captured by the video control unit24. The image memory 28 may be any image storage device, for example,one of semiconductor memories including flash memories, or one ofmagnetic disks including floppy disks and mini-disks (MD).

In the camera system of the present embodiment, a processor 30 controlsthe overall operation of the camera system and carries out a divisionalshooting process including determination of a positional relationbetween partially overlapping images based upon a common pattern in anoverlapping portion of the images. The processor 30 includes anarithmetic control unit 32, a read-only memory (ROM) 33, and a randomaccess memory (RAM) 36. The ROM 33 stores a number of programs 34Athrough 34N, and fixed information, such as character fonts. Thearithmetic control unit 32 carries out individual control operations forthe elements of the camera system when one of the programs 34A through34N in the ROM 33 is executed by the processor 30. The RAM 36 is a mainmemory of the processor 30 which is available to any of the programswhen it is executed. The RAM 36 serves as a work memory available to thearithmetic control unit 32. Further, the processor 30 includes a powersupply circuit (not shown) which supplies power to the camera system,and an interface (not shown) which connects the camera system with anexternal host computer.

In the camera system of FIG. 1, the operator can select one of theoperational modes by using the mode selection switch 16. In the presentembodiment, the operational modes of the camera system include a normalshooting mode and a divisional shooting mode.

When the normal shooting mode is selected by the mode selection switch16, a single image of an object or a scene is captured through the imagepickup device 12, the image displayed on the screen 27A of the monitor27 is viewed, and the shutter switch 20 is turned ON by the operator sothat the image defined in the frame memory 25 is stored in the imagememory 28.

When the divisional shooting mode is selected in the camera system ofthe present embodiment, a plurality of partially overlapping images aresuccessively shot so as to cover a wide angle related to an object to beimaged, and they are synthesized to create a composite image with anadequate level of resolution. The divisional shooting mode is useful toobtain a panoramic image or a high-resolution image through imagecomposition. The camera system of the present invention is particularlyrelevant to the divisional shooting mode, and the following descriptionwill be given of an operation of the camera system of the presentembodiment when the divisional shooting mode is performed.

In the camera system of the present embodiment, when the divisionalshooting mode is selected by the mode selection switch 20, the processor30 starts the execution of a divisional shooting processing program 34Iamong the programs 34A through 34N in the ROM 33. A divisional shootingprocess is performed by the processor 30 according to the divisionalshooting processing program 34I.

In order to take a first one of partially overlapping images when thedivisional shooting process is started, the operator directs the opticalaxis of the camera (or the lens of the optical unit 10) to an object tobe imaged. In accordance with the signal from the image pickup device12, the video control unit 24 stores a corresponding frame in the framememory 25, and displays the image on the screen 27A of the monitor 27.The operator turns ON the shutter switch 20 of the operation part 16while viewing the image on the screen 27A of the monitor 27. A shuttersignal from the operation part 16 is sent to the processor 30immediately after the shutter switch 20 is turned ON. In response to theshutter signal, the processor 30 stores the image, defined in the framememory 25 of the video control unit 24, in the image memory 28.

The above-mentioned image storage process is performed by the processor30 of the camera system in accordance with an image storage processingprogram 34N among the programs 34A through 34N stored in the ROM 33. Theexecution of the image storage processing program 34N is started by theprocessor 30 in response to the shutter signal. During the image storageprocess, all the image data corresponding to the entire screen 27A ofthe monitor 27 is not stored in the image memory 28, but only a portionof the image data corresponding to an internal portion of the screen 27Aof the monitor 27 within the peripheral boundary 27B is stored in theimage memory 28. The processor 30 adds a frame number to the auxiliarydata of the frame buffer 26 and stores such data defined in the framebuffer 26, in the image memory 28, together with the image defined inthe frame buffer 25, during the image storage process. The data beingstored in the image memory 28 may be compressed in a compact form or maynot be compressed in the original form. During the image storageprocess, the writing of image data to the frame buffer 25 is inhibitedand the image displayed on the screen 27A of the monitor 27 is fixed.Before the image storage process ends, the writing of image data to theframe buffer 25 is allowed. Hence, after the image storage process isperformed, the image defined in the frame buffer 25 can be variablyupdated according to the movement of the optical axis of the camera, andthe resulting image is displayed on the screen 27A of the monitor 27.

FIG. 7 shows an image storage process performed by the processor 30 ofthe camera system of the present embodiment. The image storageprocessing program 34N among the programs 34A through 34N in the ROM 33is loaded to the RAM 36 and executed by the processor 30 immediatelyafter the shutter switch 20 is turned ON by the operator. Then, theimage storage process of FIG. 7 is performed by the processor 30according to the image storage processing program 34N.

As shown in FIG. 7, at the start of the image storage process, theprocessor 30 at step S500 inhibits the writing of image data to theframe buffer 25 by the video control unit 24. Hence, during the imagestorage process, the image displayed on the screen 27A of the monitor 27is fixed.

The processor 30 at step S502 combines the auxiliary data of the framebuffer 26 with the image of the frame buffer 25 to create a synthesizedimage, and stores the synthesized image in the image memory 28. Asdescribed above, the auxiliary data of the frame buffer 26 includes aframe number to identify a captured image among the partiallyoverlapping images. The auxiliary data of the frame buffer 26 mayinclude other parameter values (which will be described later). However,when the image storage process with respect to a first one of partiallyoverlapping images is performed, the auxiliary data of the frame buffer26 is null or vacant, and only the image of the frame buffer 25 isstored in the image memory 28 at the step S502.

The processor 30 at step S504 allows the writing of image data to theframe buffer 25 by the video control unit 24. After the step S504 isperformed, the image storage process of FIG. 7 ends. Hence, after theimage storage process is performed, the image defined in the framebuffer 25 is displayed on the screen 27A of the monitor 27.

After the first one of the partially overlapping images is taken, theoperator pans the camera in a desired direction in order to take afollowing one of the partially overlapping images during the divisionalshooting mode. By viewing the preceding image with the peripheralboundary on the screen 27A of the monitor 27, the operator stops themovement of the optical axis of the camera at an appropriate positionwhere an overlapping portion of the two adjacent images is appropriatefor subsequently producing a composite image from the images. Then, thecurrent image is captured and stored in the image memory 28 in a similarmanner. The above-described procedure is repeated until all thepartially overlapping images for the object to be imaged are capturedand stored. In this manner, the partially overlapping images aresuccessively shot so as to cover a wide angle related to the object, andthey are synthesized to create a composite image with an adequate levelof resolution by using the technology as disclosed in theabove-mentioned U. S. patent applications.

According to the camera system of the present invention, the operatorcan easily carry out the divisional shooting process by viewing both thecurrent image and the peripheral boundary 27B (or the preceding image)on the screen 27A of the monitor 27. A positional relation between thepreceding image and the current image is clearly noticeable to theoperator by viewing the peripheral boundary 27B on the screen 27A of themonitor 27 and the current image while the camera is panned in thedesired direction. Therefore, the operator easily stops the movement ofthe optical axis of the camera at an appropriate position by viewing animage of the peripheral boundary 27B, and turns ON the shutter switch 20to store the current image.

FIG. 2 shows a first example of the divisional shooting processperformed by the processor 30 in accordance with the divisional shootingprocessing program 34I.

As shown in FIG. 2, at the start of the divisional shooting process, theprocessor 30 at step S100 detects whether the image storage process,shown in FIG. 7, with respect to a preceding one of the partiallyoverlapping images ends. The end of the image storage process isnotified to the arithmetic control unit 32 when the execution of theimage storage processing program 34N has normally ended. When the resultat the step S100 is negative, the processor 30 repeats the step S100.

When the result at the step S100 is affirmative, the processor 30 atstep S104 reads out the pixel map of the preceding image from the imagememory 28, and reads out the pixel map of a currently-captured imagefrom the frame buffer 25. These pixel maps are temporarily stored in theRAM 36. The pixel map of the preceding image is selected as a standardimage. Each of the pixel data of the two adjacent images correspondingto an overlapping portion of the images is divided into blocks of apredetermined size, for example, 16 by 16 pixels.

After the step S104 is performed, the processor 30 at step S106 performsa matching between corresponding blocks from an overlapping portion ofthe two adjacent images. During the step S106, a common pattern in thetwo adjacent images is identified if a certain similarity threshold ismet. This matching may be performed by checking the intensities ofindividual pixels of the corresponding blocks. This is useful forreducing the amount of required calculations. Alternatively, thematching may be performed by checking the color values of individualpixels of the corresponding blocks, but this will increase the amount ofrequired calculations. The above matching procedures are repeated untilall the blocks are processed so that a maximum-similarity common patternin the preceding image and the maximum-similarity common pattern in thecurrent image are detected.

A method and a system for determining a positional relation betweenpartially overlapping images based upon a common pattern in anoverlapping portion of the images are disclosed in the above-mentionedU. S. patent applications, and the divisional shooting process accordingto the present invention utilizes the method and the system.

As previously described, during the step S106 of the divisional shootingprocess of FIG. 2, a determination of a positional relation betweenpartially overlapping images is carried out. By referring to FIG. 8A andFIG. 8B, a detailed procedure of the determination of the positionalrelation in the step S106 will now be described.

It is supposed that the pixel map of the preceding image from the imagememory 28 and the pixel map of the current image from the frame buffer25 have been read out as in the step S104. These pixel maps aretemporarily stored in the RAM 36. The pixel map of the preceding imageis selected as the standard image. Each of the two adjacent imagescorresponding to an overlapping portion of the images is divided intoblocks of a predetermined size.

As shown in FIG. 8A, pixels “A”, “B” and “C” in the preceding image andpixels “A′”, “B′” and “C′” in the current image correspond to theoverlapping portion of the images. During the step S106, a matchingbetween corresponding blocks from the overlapping portion of the twoadjacent images is performed. A common pattern (such as the pixels A, Band C and the pixels A′, B′ and C′) in the two adjacent images isidentified if a certain similarity threshold is met. This matching maybe performed by checking the intensities of individual pixels of thecorresponding blocks. The above matching procedures are repeated untilall the blocks are processed, so that a maximum-similarity commonpattern in the preceding image and the maximum-similarity common patternin the current image are detected.

As shown in FIG. 8B, the maximum-similarity common pattern in the twoimages is detected if the difference between the pixel values (or theintensities of the pixels A and A′, the pixels B and B′ or the pixels Cand C′) of the corresponding blocks is found to be the minimum when thecurrent image is moved relative to the preceding image by both adistance for a first number of pixels in the x-axis direction and adistance for a second number of pixels in the y-axis direction. Throughthe above pixel-based method, the processor 30 detects themaximum-similarity common pattern in the two images. That is, theprocessor 30 at the step S106 carries out the determination of thepositional relation between the partially overlapping images.

In the above-described procedure, the maximum-similarity common patternin the two images is detected by using the pixel-based method, in orderto carry out the determination of the positional relation between thepartially overlapping images. However, according to the presentinvention, it is also possible to achieve the determination of apositional relation between partially overlapping images at an accuracyhigher than the accuracy of one pixel. As previously described, thedetermination of a positional relation between partially overlappingimages based upon a common pattern in an overlapping portion of theimages are disclosed in the above-mentioned U. S. patent applications,and, for that purpose, the divisional shooting process according to thepresent invention may utilize the method and the system.

Referring back to FIG. 2, during the step S106, the processor 30 furtherdetermines both coordinates (I, J) of a central pixel of themaximum-similarity common pattern in the preceding image and coordinates(Im, Jm) of a central pixel of the maximum-similarity common pattern inthe current image. The coordinates (I, J) and the coordinates (Im, Jm)based on a screen coordinate system of the screen 27A of the monitor 27are determined by the processor 30.

The processor 30 at step S108 determines a displacement vector (I-Im,J-Jm), which indicates a positional relation between the preceding imageand the current image, by the difference between the coordinates (I, J)and the coordinates (Im, Jm). In the step S108, after the contents ofthe frame buffer 26 are cleared, the processor 30 writes image data,indicative of the displacement vector, to the frame buffer 26 as part ofthe auxiliary data. Hence, the image of the displacement vector (or theauxiliary data defined in the frame buffer 26) is displayed on thescreen 27A of the monitor 27.

The processor 30 at step S110 detects whether the operator stops themovement of the optical axis of the camera (or detects whether theoperator turns ON the shutter switch 20). When the result at the stepS110 is negative, the above steps S106 and S108 are repeated.

When the step S106 is performed for second or subsequent ones of thepartially overlapping images, the coordinates (I, J) of the centralpixel of the maximum-similarity common pattern in the preceding imageand the direction of the displacement vector are known. The matchingprocedures in the step S106 may be performed for only the blocks of thecurrent image in the overlapping portion of the two images, indicated bythe direction of the displacement vector and the coordinates (I, J). Byusing such a simplified matching, the common pattern in the two adjacentimages may be identified, and coordinates (Im, Jm) of the central pixelof the maximum-similarity common pattern in the current image may bedetermined.

The operator stops the panning of the camera at an appropriate positionwhere an appropriate overlapping portion of the two adjacent images canbe seen with the image of the displacement vector on the screen 27A ofthe monitor 27, and turns ON the shutter switch 20 to store the currentimage. Every time the steps S106 and S108 are performed, the processor30 compares the currently obtained displacement vector and thepreviously obtained displacement vector (stored in an internal registerof the processor 30 or the RAM 36) so as to determine whether theoperator stops the movement of the optical axis of the camera. If thedifference between the two displacement vectors is larger than athreshold value, the result at the step S110 is negative. If thedifference between the two displacement vectors is less than thethreshold value, the result at the step S110 is affirmative.

When the result at the step S110 is affirmative, the processor 30 atstep S112 writes image data, indicative of the peripheral boundary 27Bof the preceding image, to the frame buffer 26 at a position shiftedfrom the previous position. The shifted position is determined from theprevious position based on the magnitude and direction of thedisplacement vector obtained in the step S108. Hence, the image of theperipheral boundary 27B defined in the frame buffer 26 is displayed onthe screen 27A of the monitor 27 as if the peripheral boundary 27B isshifted according to the movement of the optical axis of the camera.

In the step S112, the image data of the displacement vector obtained inthe step S108 may be left in the frame buffer 26 without change.Alternatively, the image data of the displacement vector in the framebuffer 26 may be deleted, and then the image data of the shiftedperipheral boundary 27B may be defined in the frame buffer 26. The imageof the peripheral boundary 27B being displayed on the screen 27A of themonitor 27 may be a frame of the preceding image or a solid model of thepreceding image with a certain color attached to the internal pixels.

The operator can easily carry out the divisional shooting process withthe camera system by viewing both the current image and the peripheralboundary 27B (or the preceding image) on the screen 27A of the monitor27. A positional relation between the preceding image and the currentimage is clearly noticeable to the operator by viewing the peripheralboundary 27B on the screen 27A of the monitor 27 and the current imagewhile the camera is panned in a desired direction. Therefore, theoperator easily stops the movement of the optical axis of the camera atan appropriate position by viewing an image of the peripheral boundary27A, and turns ON the shutter switch 20 to store the current image.

After the step S112 is performed, the control is transferred to the stepS100. The processor 30 at the step S100 waits for the end of the imagestorage process at which the currently captured image is further storedin the image memory 28. As described above, during the image storageprocess, the frame number for the current image and the displacementvector for the current image are added to the auxiliary data of theframe buffer 26 and such data defined in the frame buffer 26 is storedin the image memory 28 together with the image defined in the framebuffer 25. The frame number and the displacement data are used whensynthesizing the partially overlapping images to create a compositeimage.

FIG. 3A shows an image which is displayed on the screen 27A of themonitor 27 when the camera is being moved in a given direction indicatedin FIG. 3A. In FIG. 3A, a peripheral boundary of a preceding image isindicated by the dotted-line rectangle A′B′C′D′, and a peripheralboundary of a current image is indicated by the solid-line rectangleABCD. A displacement between the preceding image and the current imageproportional to the movement of the optical axis of the camera isdefined by the displacement vector. In the case of FIG. 3A, thedisplacement vector is directed to the left and has a lengthproportional to the movement of the optical axis of the camera. An image50 of the displacement vector is displayed on the screen 27A of themonitor 27 as indicated in FIG. 3A. Although the contents of thepreceding image are not displayed, the operator can easily notice apositional relation between the preceding image and the current image onthe screen 27A of the monitor 27 with the image 50.

FIG. 3B shows an image which is displayed on the screen 27A of themonitor 27 when the movement of the optical axis of the camera isstopped and the shutter switch 20 is turned ON by the operator. In FIG.3B, an image 52 of the peripheral boundary 27B, which is displayed onthe screen 27A of the monitor 27, is indicated by the rectangle ABC′D′.The rectangle ABC′D′ corresponds to an overlapping portion of the twoadjacent images. As described above, the image data, indicative of theperipheral boundary 27B of the preceding image, is written to the framebuffer 26 at positions shifted from the previous positions according tothe movement of the optical axis of the camera. The image 50 of thedisplacement vector corresponding to the magnitude and direction of thedisplacement vector is displayed on the screen 27A of the monitor 27.The operator can clearly notice an appropriate overlapping portion ofthe two images by the image 50 of the displacement vector and the image52 of the peripheral boundary 27B. The image 50 of the displacementvector, at the time the movement of the optical axis of the camera isstopped, may be displayed on the screen 27A of the monitor 27.Alternatively, the display of the image 50 of the displacement vectormay be omitted.

FIG.4 shows a second example of the divisional shooting processperformed by the processor 30 in accordance with the divisional shootingprocessing program 34I.

As shown in FIG. 4, at the start of the divisional shooting process inthe present embodiment, the processor 30 at step S200 detects whetherthe image storage process with respect to a preceding one of thepartially overlapping images ends. The end of the image storage processis notified to the arithmetic control unit 32 when the execution of theimage storage processing program 34N has normally ended. When the resultat the step S200 is negative, the processor 30 repeats the step S200.

When the result at the step S200 is affirmative, the processor 30 atstep S204 reads out the pixel map of the preceding image from the imagememory 28, and reads out the pixel map of the currently-captured imagefrom the frame buffer 25. The pixel maps are temporarily stored in theRAM 36. The pixel map of the preceding image is selected as a standardimage. Each of the pixel data of the two adjacent images correspondingto the overlapping portion of the images is divided into blocks of apredetermined size, for example, 16 by 16 pixels.

After the step S204 is performed, the processor 30 at step S206 performsa matching between corresponding blocks from the two adjacent images.During the step S206, a common pattern in the two adjacent images isidentified if a certain similarity threshold is met. The matchingprocedures are repeated for every block until all the blocks areprocessed so that the common pattern in the preceding image and thecommon pattern in the current image are identified.

Further, during the step S206, the processor 30 determines bothcoordinates (I, J) of a central pixel of a maximum-similarity commonpattern in the preceding image and coordinates (Im, Jm) of a centralpixel of the maximum-similarity common pattern in the current image. Thecoordinates (I, J) and the coordinates (Im, Jm) based on a screencoordinate system of the screen 27A of the monitor 27 are determined bythe processor 30.

The steps S200-S206 in the present embodiment are essentially the sameas the steps S100-S106 in the embodiment of FIG. 2.

The processor at step S208 determines a displacement vector (I-Im,J-Jm), which indicates a positional relation between the preceding imageand the current image, by the difference between the coordinates (I, J)and the coordinates (Im, Jm). In the present embodiment, during the stepS208, the processor 30 writes image data, indicative of the peripheralboundary 27B of the preceding image, to the frame buffer 26 at positionsshifted from the previous positions. The shifted positions are indicatedby the magnitude and direction of the displacement vector. Hence, theimage of the peripheral boundary 27B defined in the frame buffer 26 isdisplayed on the screen 27A of the monitor 27.

Unlike the embodiment of FIG. 2, during the step S208 in the presentembodiment, the processor 30 does not write the image data of thedisplacement vector to the frame buffer 26 as part of the auxiliarydata. Hence, the image of the displacement vector is not displayed onthe screen 27A of the monitor 27.

The processor 30 at step S210 detects whether the operator stops themovement of the optical axis of the camera (or detects whether theoperator turns ON the shutter switch 20). When the result at the stepS210 is negative, the above steps S206 and S208 are repeated.

The operator stops the panning of the camera at an appropriate positionwhere an appropriate overlapping portion of the two adjacent images canbe seen with the image of the displacement vector on the screen 27A ofthe monitor 27, and turns ON the shutter switch 20 to store the currentimage. Every time the steps S206 and S208 are performed, the processor30 compares the currently obtained displacement vector and thepreviously obtained displacement vector (stored in the internal registeror the RAM 36) so as to determine whether the operator stops the panningof the camera. If the difference between the two displacement vectors islarger than a threshold value, the result at the step S210 is negative.If the difference between the two displacement vectors is less than thethreshold value, the result at the step S210 is affirmative.

When the result at the step S210 is affirmative, the control istransferred to the step S200. The processor 30 at the step S200 waitsfor the end of the image storage process at which the currently capturedimage is further stored in the image memory 28.

In the present embodiment, the operator can view a peripheral boundaryimage indicating a positional relation between the current image and thepreceding image before the movement of the optical axis of the camera isstopped or the shutter switch 20 is turned ON. The operator can easilycarry out the divisional shooting process with the camera system, butthe current image and the peripheral boundary image are always displayedon the screen 27A of the monitor 27. It is desirable that the intensityand/or color of the peripheral boundary image may be set at a suitablevalue so as to prevent the peripheral boundary image from hindering thecheck for the current image on the screen 27A of the monitor 27.

FIG. 5 shows a third example of the divisional shooting processperformed by the processor 30 in accordance with the divisional shootingprocessing program 34I.

In the present embodiment, the camera system further includes athree-dimensional gyro sensor 40 connected to the arithmetic controlunit 32 of the processor 30 as indicated by the dotted line in FIG. 1.The sensor 40 detects a three-dimensional direction of the optical axisof the optical unit 10 and outputs a signal indicating the optical axisdirection to the arithmetic control unit 32 of the processor 30. Thesensor 40 may be a built-in type or an external-installation type forthe camera system. Other elements of the camera system in the presentembodiment are the same as corresponding elements of the camera systemshown in FIG. 1, and a description thereof will be omitted.

When the divisional shooting mode is selected by the mode selectionswitch 20, the processor 30 starts the execution of the divisionalshooting processing program 34I in the ROM 33. The present embodiment ofthe divisional shooting process is performed by the processor 30according to the divisional shooting processing program 34I.

In order to take a first one of partially overlapping images at thestart of the divisional shooting process is started, the operatordirects the optical axis of the camera to an object to be imaged andturns ON the shutter switch 20. A shutter signal from the operation part16 is sent to the processor 30 immediately after the shutter switch 20is turned ON. In response to the shutter signal, the processor 30 readsa signal output by the sensor 40 at that time, and temporarily storesthe signal in an internal register of the processor 30 or the RAM 36. Inaccordance with the signal from the image pickup device 12, the videocontrol unit 24 stores a corresponding frame in the frame memory 25, anddisplays the image on the screen 27A of the monitor 27. In response tothe shutter signal, the processor 30 stores the image, defined in theframe memory 25, in the image memory 28.

The above-mentioned image storage process is performed by the processor30 according to the image storage processing program 34N in the ROM 33.The execution of the image storage processing program 34N is started bythe processor 30 in response to the shutter signal. During the imagestorage process, the processor 30 adds both the frame number and theoptical axis direction signal to the auxiliary data of the frame buffer26, and stores such data defined in the frame buffer 26, in the imagememory 28, together with the image defined in the frame buffer 25.During the image storage process, the writing of image data to the framebuffer 25 is inhibited and the image displayed on the screen 27A of themonitor 27 is fixed. Before the image storage process ends, the writingof image data to the frame buffer 25 is allowed. Hence, after the imagestorage process is performed, the image defined in the frame buffer 25can be variably updated according to the movement of the optical axis ofthe camera, and the resulting image is displayed on the screen 27A ofthe monitor 27.

After the first one of the partially overlapping images is taken, theoperator pans the camera in a desired direction in order to take afollowing one of the partially overlapping images during the divisionalshooting mode. By viewing the preceding image with the peripheralboundary on the screen 27A of the monitor 27, the operator stops themovement of the optical axis of the camera such that the preceding imageand the currently-captured image overlap each other with an appropriateoverlapping portion of the images. Then, the current image is capturedand stored in the image memory 28 together with the auxiliary data,including the frame number and the optical axis direction signal, in asimilar manner. The above-described procedure is repeated until all thepartially overlapping images for the object to be imaged are capturedand stored.

With reference to FIG. 5, a description will now be given of the thirdexample of the divisional shooting process performed by the processor30.

As shown in FIG. 5, at the start of the divisional shooting process inthe present embodiment, the processor 30 at step S300 detects whetherthe image storage process of FIG. 7 with respect to a preceding one ofthe partially overlapping images ends. The end of the image storageprocess is notified to the processor 30. When the result at the stepS300 is negative, the processor 30 repeats the step S300.

When the result at the step S300 is affirmative, the processor 30 atstep S304 reads an optical axis direction signal (related to the currentimage) output by the sensor 40 at that time, and reads the optical axisdirection signal (related to the preceding image) from the internalregister or the RAM 36.

After the step S304 is performed, the processor 30 at step S306determines a displacement vector, which indicates a positional relationof the preceding image to the current image on the screen 27A of themonitor 27, by the difference between the optical axis direction signalrelated to the preceding image and the optical axis direction signalrelated to the current image.

The processor 30 at step S308 writes image data, indicative of thedisplacement vector, to the frame buffer 26 as part of the auxiliarydata after the contents of the frame buffer 26 are cleared. Hence, animage of the displacement vector (or the auxiliary data defined in theframe buffer 26) is displayed on the screen 27A of the monitor 27,similar to the image 50 shown in FIG. 3A and FIG. 3B.

The processor 30 at step S310 detects whether the operator stops themovement of the optical axis of the camera (or detects whether theoperator turns ON the shutter switch 20). When the result at the stepS310 is negative, the above steps S304 through S308 are repeated.

The operator stops the panning of the camera at an appropriate positionwhere an appropriate overlapping portion of the two adjacent images canbe seen with the image of the displacement vector on the screen 27A ofthe monitor 27, and turns ON the shutter switch 20 to store the currentimage. Every time the steps S304 through S308 are performed, theprocessor 30 compares the currently obtained displacement vector and thepreviously obtained displacement vector (stored in the internal registeror the RAM 36) so as to determine whether the operator stops themovement of the optical axis of the camera. If the difference betweenthe two displacement vectors is larger than a threshold value, theresult at the step S310 is negative. If the difference between the twodisplacement vectors is less than the threshold value, the result at thestep S310 is affirmative.

When the result at the step S310 is affirmative, the processor 30 atstep S312 writes image data, indicative of the peripheral boundary 27Bof the preceding image, to the frame buffer 26 at positions shifted fromthe previous positions. The shifted positions are indicated by themagnitude and direction of the displacement vector obtained in the stepS306. Hence, the image of the peripheral boundary 27B defined in theframe buffer 26 is displayed on the screen 27A of the monitor 27.

In the step S312, the image data of the displacement vector obtained inthe step S306 may be left in the frame buffer 26 without change.Alternatively, the image data of the displacement vector in the framebuffer 26 may be deleted, and then the image data of the shiftedperipheral boundary 27B may be defined in the frame buffer 26. The imageof the peripheral boundary 27B being displayed on the screen 27A of themonitor 27 may be a frame of the preceding image or a solid model of thepreceding image with a certain color attached to the internal pixels.

The operator can easily carry out the divisional shooting process withthe camera system by viewing both the current image and the peripheralboundary 27B (or the preceding image) on the screen 27A of the monitor27. A positional relation between the preceding image and the currentimage is clearly noticeable to the operator by viewing the peripheralboundary 27B on the screen 27A of the monitor 27 and the current imagewhile the camera is panned in a desired direction. Therefore, theoperator easily stops the movement of the optical axis of the camera atan appropriate position by viewing an image of the peripheral boundary27B, and turns ON the shutter switch 20 to store the current image.

After the step S312 is performed, the control is transferred to the stepS300. The processor 30 at the step S300 waits for the end of the imagestorage process at which the currently captured image is further storedin the image memory 28. As described above, during the image storageprocess, the frame number for the current image and the displacementvector for the current image are added to the auxiliary data of theframe buffer 26 and such data defined in the frame buffer 26 is storedin the image memory 28 together with the image defined in the framebuffer 25. The frame number and the displacement data are used whensynthesizing the partially overlapping images to create a compositeimage.

FIG. 6 shows a fourth example of the divisional shooting processperformed by the processor 30 in accordance with a divisional shootingprocessing program 34I.

As shown in FIG. 6, at the start of the divisional shooting process inthe present embodiment, the processor 30 at step S400 detects whetherthe image storage process of FIG. 7 with respect to a preceding one ofthe partially overlapping images ends. The end of the image storageprocess is notified to the processor 30. When the result at the stepS400 is negative, the processor 30 repeats the step S400.

When the result at the step S400 is affirmative, the processor 30 atstep S404 reads an optical axis direction signal (related to the currentimage) output by the sensor 40 at that time, and reads the optical axisdirection signal (related to the preceding image) from the internalregister or the RAM 36.

After the step S404 is performed, the processor 30 at step S406determines a displacement vector, which indicates a positional relationof the preceding image to the current image on the screen 27A of themonitor 27, by the difference between the optical axis direction signalrelated to the preceding image and the optical axis direction signalrelated to the current image.

The processor 30 at step S408 writes image data, indicative of theperipheral boundary 27B of the preceding image, to the frame buffer 26at positions shifted from the previous positions. The shifted positionsare indicated by the magnitude and direction of the displacement vectorobtained in the step S406. Hence, an image of the peripheral boundary27B defined in the frame buffer 26 is displayed on the screen 27A of themonitor 27, similar to the image 52 shown in FIG. 3B.

The processor 30 at step S410 detects whether the operator stops themovement of the optical axis of the camera (or detects whether theoperator turns ON the shutter switch 20). When the result at the stepS410 is negative, the above steps S404 through S408 are repeated.

The operator stops the panning of the camera at an appropriate positionwhere an appropriate overlapping portion of the two adjacent images canbe seen with the image of the peripheral boundary 27B on the screen 27Aof the monitor 27, and turns ON the shutter switch 20 to store thecurrent image. Every time the steps S404 through S408 are performed, theprocessor 30 compares the currently obtained displacement vector and thepreviously obtained displacement vector (stored in the internal registeror the RAM 36) so as to determine whether the operator stops themovement of the optical axis of the camera. If the difference betweenthe two displacement vectors is larger than a threshold value, theresult at the step S410 is negative. If the difference between the twodisplacement vectors is less than the threshold value, the result at thestep S410 is affirmative.

The operator can easily carry out the divisional shooting process withthe camera system by viewing both the current image and the peripheralboundary 27B (or the preceding image) on the screen 27A of the monitor27. A positional relation between the preceding image and the currentimage is clearly noticeable to the operator by viewing the peripheralboundary 27B on the screen 27A of the monitor 27 and the current imagewhile the camera is panned in a desired direction. Therefore, theoperator easily stops the movement of the optical axis of the camera atan appropriate position by viewing the image of the peripheral boundary27B, and turns ON the shutter switch 20 to store the current image.

When the result at the step S410 is affirmative, the control istransferred to the step S400. The processor 30 at the step S400 waitsfor the end of the image storage process at which the currently capturedimage is further stored in the image memory 28.

The above-described embodiments of the present invention are applied toa digital camera. However, the present invention is not limited to theabove-described embodiments. It is readily understood that the presentinvention is essentially applicable to a still-video camera and othercamera systems which electronically store an image of an object anddisplay the image on a display monitor. Further, variations andmodifications of the above-described embodiments may be made withoutdeparting from the scope of the present invention.

The present invention is based on Japanese priority application No.9-245522, filed on Sep. 10, 1997, the entire contents of which arehereby incorporated by reference.

What is claimed is:
 1. A camera system comprising: a display monitor fordisplaying an image of an object, taken by an optical unit and stored ina frame buffer, on a screen of the monitor; a reading unit for reading apreceding image and a current image among a plurality of partiallyoverlapping images, from the frame buffer, the preceding image and thecurrent image containing a common element; a determining unit fordetermining a positional relation between the preceding image and thecurrent image based on a common pattern derived from the common elementin the two adjacent images read by the reading unit; and a displayingunit for displaying, when a shutter switch is turned on, a displayingunit image comprising the concurrent display of (i) a boundary imageindicating a boundary of the preceding image on the screen of themonitor at a shifted position according to the positional relationdetermined by the determining unit, and (ii) the current image, whereinone of the plurality of partially overlapping images is defined by theoverlap of the boundary image and the current image.
 2. The camerasystem according to claim 1, wherein the determining unit performs amatching between corresponding blocks taken from an overlapping portionof the two adjacent images, so that a maximum-similarity common patternin the two adjacent images is identified.
 3. The camera system accordingto claim 1, wherein the determining unit performs a matching betweencorresponding blocks taken from an overlapping portion of the twoadjacent images by checking intensities of individual pixels of thecorresponding blocks.
 4. The camera system according to claim 1, whereinthe determining unit determines both coordinates of a central pixel of amaximum-similarity common pattern in the preceding image and coordinatesof a central pixel of the maximum-similarity common pattern in thecurrent image.
 5. The camera system according to claim 1, furthercomprising: a sensor for outputting an optical axis direction signalindicating a direction of an optical axis of the optical unit; and asecondary determining unit for determining a positional relation betweenthe preceding image and the current image based on a difference betweenthe optical axis direction signal output by the sensor with respect tothe current image and the optical axis direction signal output by thesensor with respect to the preceding image.
 6. The camera systemaccording to claim 1, wherein the determining unit determines adisplacement vector, indicating a positional relation between thepreceding image and the current image, based on a difference betweencoordinates of a central pixel of a maximum-similarity common pattern inthe preceding image and coordinates of a central pixel of themaximum-similarity common pattern in the current image, and wherein thedisplaying unit displays an image of the displacement vector on thescreen of the monitor with the current image concurrently displayed onthe screen of the monitor.
 7. The camera system according to claim 1,further comprising an image storing unit for storing an image of theobject, taken by the optical unit, in an image memory, wherein the imagestoring unit stores auxiliary data, containing information indicatingthe positional relation from the determining unit, in the image memory,in addition to the image stored therein.
 8. The camera system accordingto claim 5, further comprising an image storing unit for storing animage of the object, taken by the optical unit, in an image memory,wherein the image storing unit stores auxiliary data, containinginformation indicating the positional relation from the secondarydetermining unit, in the image memory, in additional to the image storedtherein.
 9. A divisional shooting method for a camera system in which atleast two of partially overlapping images of an object, taken by anoptical unit and stored in a frame buffer, are displayed, comprising thesteps of: reading a preceding image and a current image among thepartially overlapping images, from the frame buffer, the preceding imageand the current image containing a common element; determining apositional relation between the preceding image and the current imagebased on a common pattern derived from the common element in the twoadjacent images; and displaying, when a shutter switch is turned on, adisplaying unit image comprising the concurrent display of (i) aboundary image indicating a boundary of the preceding image on a screenof a display monitor at a shifted position according to the positionalrelation determined by the determining step, and (ii) the current image,wherein one of the at least two partially overlapping images is definedby the overlap of the boundary image and the current image.
 10. Themethod according to claim 9, wherein, in the determining step, amatching between corresponding blocks taken from an overlapping portionof the two adjacent images is performed, so that a maximum-similaritycommon pattern in the two adjacent images is identified.
 11. The methodaccording to claim 9, wherein, in the determining step, a matchingbetween corresponding blocks taken from an overlapping portion of thetwo adjacent images is performed by checking intensities of individualpixels of the corresponding blocks.
 12. The method according to claim 9,wherein, in the determining step, both coordinates of a central pixel ofa maximum-similarity common pattern in the preceding image andcoordinates of a central pixel of the maximum-similarity common patternin the current image are determined.
 13. The method according to claim9, further comprising the steps: outputting an optical axis directionsignal indicating a direction of an optical axis of the optical unit;and determining a positional relation between the preceding image andthe current image based on a difference between the optical axisdirection signal output by the sensor with respect to the current imageand the optical axis direction signal output by the sensor with respectto the preceding image.
 14. The method according to claim 9, wherein, inthe determining step, a displacement vector, indicating a positionalrelation between the preceding image and the current image, isdetermined based on a difference between coordinates of a central pixelof a maximum-similarity common pattern in the preceding image andcoordinates of a central pixel of the maximum-similarity common patternin the current image, and wherein, in the displaying step, an image ofthe displacement vector is displayed on the screen of the monitor withthe current image concurrently displayed on the screen of the monitor.15. The method according to claim 9, further comprising a step ofstoring an image of the object, taken by the optical unit, in an imagememory, wherein auxiliary data, containing information indicating thepositional relation from the determining unit, is stored in the imagememory in addition to the image stored therein.
 16. The method accordingto claim 13, further comprising a step of storing an image of theobject, taken by the optical unit, in an image memory, wherein auxiliarydata, containing information indicating the positional relation from thesecondary determining unit, is stored in the image memory, in additionalto the image stored therein.
 17. The camera system according to claim 1,wherein the displaying unit displays a displacement vector,corresponding to the positional relation between the preceding image andthe current image, during movement of an optical axis of the opticalunit.
 18. The method according to claim 9, further comprising the stepof displaying a displacement vector, corresponding to the positionalrelation between the preceding image and the current image, duringmovement of an optical axis of the optical unit.